1. The Field of the Invention
This invention relates to methods and systems for computerized authoring, learning, and evaluation. More specifically, this invention relates to methods and systems for allowing an author to organize and present information to a student and to generate test questions contextually relevant to the learned material.
2. The Relevant Technology
Testing has typically held a central role in the learning process. In educational settings, an individual is first presented with information that is to be learned and, in order to measure the amount of information learned, a test is given. Testing has evolved into a specialized field. Many theories have been developed in order to both develop tests designed to measure a particular level of achievement or mastery and to interpret the results of such a test. Thus, standardized tests have been developed to measure such things as Intelligence Quotient, personality traits, fitness to practice a profession such as law, medicine, contracting or the like, aptitude for success in a specific environment, and mastery of individual skills.
Historically, tests have been administrated on either an individual basis, such as oral or written examination, or on a group basis, such as a test administered in an educational setting where students record their answers on paper which are to be evaluated at a later time. One of the advantages of a written test is the efficiency in testing a large number of individuals. Unfortunately, a written test administered in such a manner lacks the ability to provide immediate feedback on an individual basis. Thus, it is generally recognized that testing in a traditional educational setting represents a tradeoff between efficiency and the ability to provide individual feedback.
It is generally unreasonable to expect that an individual test would be developed for each individual. Thus, in order to preserve the ability of the test to measure an individual's aptitude, care must be taken not to disclose the answers to a test. Likewise, where the test is designed to measure individual performance, care must be taken to insure that each individual takes the test based on his or her own knowledge.
In order to prevent cheating and to minimize the probability of an individual test score being influenced by others, multiple tests may be developed and administered simultaneously so that each individual is, in effect, taking a different test. Problems associated with this methodology, however, generally prevent its implementation. The effort and expense of developing and grading multiple tests generally limits this methodology to very specialized applications.
Many educators believe that measurement of performance and feedback is critical in the learning process. While written tests may be an efficient way to measure performance, they are largely inefficient in providing adequate feedback for learning. Generally there is a large lag time between the time when the test is taken and when the results of the test are returned. During this lag time, new information is usually presented. Thus, when the tests are returned, the focus of both the material and the individual has shifted. Thus, relatively few students will use such a test to evaluate their weaknesses and return to prior material in order to strengthen those weaknesses.
In order to overcome this limitation, some courses of study are structured with many smaller tests interspersed throughout the material. While this generally improves the potential for feedback, still such a method is relatively inefficient. Further, it dramatically increases the workload of an instructor or teacher.
In order to enhance the learning environment, many educators realize that the optimal form of instruction would be a personalized instructor for each student or individual. This, however, is generally impractical in most learning environments. In order to provide a better learning environment that more closely approaches the ideal environment of one instructor for every student, some educators are turning to computers. Through the use of computers, learning programs may be developed which provide instruction and feedback virtually simultaneously. For example, a general course of instruction may be presented to an individual after which the computer can query the individual regarding the principles just learned. The computer can then tally the score and provide the score to the individual. This allows the individual to return to information not learned and again review the material. Another advantage of computers is that they allow the pace of instruction to be varied according to the ability of the individual to learn. Furthermore, computers can be used to enhance personal learning outside the traditional educational environment. Computers thus hold great potential for enhancing the learning environment.
Current utilization of computers as part of the learning environment take their modes and methods of operation from current teaching techniques. Most computer programs designed to aid in the learning process first present a section of information and then test the individual based on the information presented. The structures of these programs are generally organized in a pre-set or pre-defined manner. Thus, like text books generally utilized in educational settings, the program presents a chapter of information and then quizzes the user on the information contained in that chapter. The next chapter is then presented, if any, along with the associated test. Programs which are organized to present information in this manner, are generally very inflexible and do not allow the user to determine in which order the information will be presented. Although this has the advantage of presenting information in an order which has been shown by long experience to be the "best" for learning the information, it forces the user to review information that may already be known.
In order to provide more flexibility, some programs allow the user to select which chapters will be presented. Thus, the user is allowed to determine the basic ordering of the chapters of information. While this allows the user some degree of flexibility in the process, it still deals with information in relatively large units. Thus, the presentation of information still remains relatively fixed and ridged. Concepts from different units or chapters must be tied together in the mind of the user. Similarly, because the tests are presented after each unit, no overview test is provided. It would thus be an advancement in the art to provide a method to allow the user more control not only over the order and sequence of presentation but also the focus and scope of the tests administered.
In an attempt to remedy this shortcoming, some programs are also provided with a global or overview test. These tests allow a user to be tested on the entire subject matter Because of the general nature of these tests, a user must generally review all material in the program before these tests can be utilized. Thus, there still remains an inability to test only on the information that has been presented to a user.
Another potential problem with these types of programs is the inability of the program to vary its behavior when presenting test questions. Once an individual has worked through a lesson or chapter, the set of questions at the end are generally fixed. Thus, an individual who must review the material several times in order to learn it, may learn the sequence of test questions. The test then becomes less a measure of the actual knowledge possessed by the user and more a measure of the ability of the user to memorize the sequence and order of test questions. In order to prevent this occurrence, many programs have built into them a random component. This component presents the test questions in a random order. Memorization of the test questions by an individual is thus more difficult. While this has the advantage of providing a little different look each time the program is run, it still does not totally solve the problem.
In an attempt to strengthen this part of the program, certain manufactures have created a larger set of questions. The test would then ask a subset of these questions in a random order. In this way, not only is the order of test questions randomized, but also the selection of the test questions themselves. Thus, a user reviewing the same material on two different occasions, may not be presented with the same questions. While this provides an even greater degree of randomness so that the tests are different each time they are taken, care must be utilized to insure that each question is asked only once. Furthermore, the effectiveness of this method depends on the number of questions available for use relative to the number of questions which are asked during each test.
This problem raises one of the limitations of these types of programs. The effectiveness of the test and evaluation section of the program is generally dependent on the number of questions that can be asked of an individual. During development of these programs, it is often very difficult and time consuming for an individual to generate a pool of questions for use during testing. Each question must be authored and then entered and programmed into the computer so that it can be asked at the appropriate time. The larger the number of test questions, the harder the program is to develop. Thus, many of these type of programs utilize relatively simple testing methodologies in order to minimize the total number of test questions that must be developed.
Speed and complexity of development are always paramount considerations when crafting these types of learning programs. Development of these types of programs generally begins by obtaining information. After information is obtained, the information must be sorted into logical categories for presentation. This process can be illustrated by a glance through a typical classroom textbook. In the textbook, information is organized or grouped by chapters to provide a logical and coherent flow for the individual utilizing the textbook. In much the same way, information is organized and structured into chapters or units or lessons to facilitate this logical and orderly flow. For each chapter or lesson, if testing is desired, then a pool of test questions must be drafted and developed. The answers to these questions must also be developed. The programmer or developer then organizes the information so that it can be presented to the user. The test questions and answers are also organized so they can be presented to the user.
The time it takes to develop a learning program can be very significant. In order to minimize the development time, there is a tendency to minimize either the amount of information presented or the number of test questions presented. This, however, is sometimes not acceptable since the resulting product is insufficient to present the depth or breadth of information required. Fore example, only questions dealing with the narrow range of presented information may be asked. The student is not presented with a broad enough spectrum of information to allow the answering of analogous rather than specific questions. Currently, there does not exist a satisfactory method of reducing development time. While some authoring tools exist, these are generally structured so as to minimize the effort involved in presenting information to the user. Test questions and testing methods must still be developed through a very laborious process. Thus, it would be an advancement in the art to have a method which reduces the effort required to develop test questions.
Recently, authoring tools have been implemented that provide for the automatic generation of questions that is related to the lesson material authored. One such method is found in U.S. Pat. No. 5,797,753, issued Aug. 25, 1998, which describes an authoring tool that allows the program to automatically generate question sets to evaluate a learner's review of a subject matter. Unfortunately, the question sets, although designed to be objective in evaluating the information learned by the student, suffer several weaknesses. The first weakness was that the content of the material presented to the student was not placed in any type of contextual relation. The lack of contextual relation turned the experiment merely into a rote learning experience for the student without any logical connection to the overall subject matter of the given topic. Contextual relationships typically pull together the subject matter in a satisfying and useful whole for the student to understand.
Another disadvantage involved the question set generation of a right answer and various wrong answers or distractors. The authoring tool had no problem generating the correct right answer associated with the generated question; however, the distractors were typically selected arbitrarily from the resource material presented to the student without consideration to the context of the question asked. Since the distractors were selected arbitrarily, they did not always have a contextual relation to the question and thus could be eliminated rather quickly as being irrelevant. Accordingly, if all the distractors were selected in such a fashion, then only the right answer would have any sense of relation to the question generated and no fair evaluation of the student's true understanding of the subject matter would be achieved.
Yet another disadvantage of the prior evaluation system is that without the contextual relationships of the underlying subject matter the author was at the mercy of the question generation portion to generate question sets that may provide a fair evaluation of the underlying subject matter. As the question sets would be generated in an arbitrary fashion, it was not always possible to provide specific questions that were necessary to test the student's actual understanding of critical subject matter within the given lesson as required by the author/instructor. In other words, the tests were not standard or fair in evaluating similiar understanding between the different students as the test questions were too random to provide.
Accordingly, what is needed is an authoring tool and tutoring component of that tool that allows the author to provide contextual relations of the underlying subject matter. The contextual relations are then used in the generation of the question sets so that the predicate question has direct relation to the right answer as would all the distracting wrong answers associated with the predicate. Furthermore, what is needed is an authoring tool and question generation mechanism of the authoring tool to provide the author with the ability to review the questions prior to submission to the student, so that the author can verify that a minimum core level of testing and evaluation is being performed by the questions presented. Furthermore, what is needed is a question generation system within an authoring tool that accurately generates questions on topic where all relevant portions of the possible answers to the question have contextual relevance to the predicate question.